Performance Assessments From the Perspective of Generalizability Theory

Historically, reliability issues in psychology and education have been addressed principally using classical test theory (CTT), which postulates that an observed score can be decomposed into a “true” score and a single, undifferentiated random error term, E (Feldt & Brennan, 1989). Generalizability (G) theory liberalizes CTT by providing models and methods that allow an investigator to disentangle multiple sources of error that contribute to E. This is accomplished, in part, through the application of certain analysis of variance (ANOVA) methods. The purposes of this paper are: (1) to provide an overview of those aspects of the conceptual framework and methods of G theory that are particularly relevant to performance assessments (PAs), (2) to evaluate some of the more prevalent estimates of reliability for PAs from the perspective of G theory, and (3) to discuss the role of various facets in assessing the generalizability of PAs.